from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting("onnx", config="config.yml")
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: algorithm=brute.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	NaN	6.759	0.0	-1	1	NaN	20.236	0.109	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	NaN	6.757	0.0	-1	5	NaN	0.350	0.011	0.034	0.034	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.001	NaN	6.318	0.0	1	100	NaN	20.057	0.032	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	NaN	6.683	0.0	-1	100	NaN	0.354	0.011	0.034	0.034	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	NaN	6.788	0.0	1	5	NaN	4.288	0.025	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	NaN	6.747	0.0	1	1	NaN	0.282	0.005	0.042	0.042	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	NaN	0.304	0.0	-1	1	NaN	4.289	0.041	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	NaN	0.318	0.0	-1	5	NaN	0.282	0.007	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.065	0.151	NaN	0.000	0.002	-1	1	0.663	0.348	0.011	5.939	5.942	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.002	NaN	0.000	0.025	-1	1	1.000	20.147	0.035	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.080	0.044	NaN	0.000	0.003	-1	5	0.757	19.985	0.041	0.154	0.154	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.029	0.006	NaN	0.000	0.029	-1	5	1.000	0.351	0.010	0.083	0.083	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.300	0.010	NaN	0.000	0.002	1	100	0.882	0.353	0.011	6.508	6.512	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.000	NaN	0.000	0.022	1	100	1.000	19.948	0.023	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.117	0.046	NaN	0.000	0.003	-1	100	0.882	19.918	0.173	0.156	0.156	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.002	NaN	0.000	0.024	-1	100	1.000	0.352	0.009	0.069	0.069	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.269	0.012	NaN	0.000	0.002	1	5	0.757	0.282	0.007	8.058	8.060	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.000	NaN	0.000	0.022	1	5	1.000	4.205	0.025	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.240	0.007	NaN	0.001	0.001	1	1	0.663	4.284	0.031	0.289	0.289	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.004	NaN	0.000	0.022	1	1	1.000	0.283	0.005	0.078	0.078	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.795	0.034	NaN	0.000	0.002	-1	1	0.896	0.278	0.005	6.452	6.453	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.005	0.001	NaN	0.000	0.005	-1	1	1.000	4.214	0.031	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.881	0.043	NaN	0.000	0.003	-1	5	0.922	4.293	0.022	0.671	0.671	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.011	0.002	NaN	0.000	0.011	-1	5	1.000	0.283	0.005	0.037	0.037	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: algorithm=kd_tree.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.418	0.048	NaN	0.023	0.0	-1	1	NaN	123.460	0.000	0.028	0.028	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.132	0.063	NaN	0.019	0.0	-1	5	NaN	3.016	0.178	1.370	1.372	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.962	0.074	NaN	0.020	0.0	1	100	NaN	121.199	0.000	0.033	0.033	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.117	0.058	NaN	0.019	0.0	-1	100	NaN	3.039	0.200	1.355	1.358	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.203	0.057	NaN	0.019	0.0	1	5	NaN	0.052	0.015	80.526	83.691	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.154	0.092	NaN	0.019	0.0	1	1	NaN	0.006	0.000	657.979	658.810	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	NaN	0.019	0.0	-1	1	NaN	0.071	0.000	0.012	0.012	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	NaN	0.026	0.0	-1	5	NaN	0.006	0.000	0.102	0.102	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.855	1.003	NaN	0.000	0.001	-1	1	0.929	3.055	0.216	0.280	0.280	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	-1	1	1.000	120.411	0.000	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.064	0.323	NaN	0.000	0.001	-1	5	0.946	120.987	0.000	0.009	0.009	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.000	NaN	0.000	0.004	-1	5	1.000	3.031	0.186	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.744	0.437	NaN	0.000	0.006	1	100	0.951	3.081	0.198	1.865	1.868	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	NaN	0.000	0.004	1	100	1.000	121.696	0.000	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.477	0.282	NaN	0.000	0.003	-1	100	0.951	122.477	0.000	0.028	0.028	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.006	0.001	NaN	0.000	0.006	-1	100	1.000	3.081	0.221	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.781	0.320	NaN	0.000	0.002	1	5	0.946	0.006	0.000	292.061	292.124	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	NaN	0.000	0.002	1	5	1.000	0.045	0.001	0.036	0.036	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.988	0.291	NaN	0.000	0.001	1	1	0.929	0.072	0.001	13.660	13.662	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	NaN	0.000	0.001	1	1	1.000	0.006	0.000	0.177	0.177	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.032	0.013	NaN	0.000	0.000	-1	1	0.891	0.006	0.000	5.252	5.254	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	NaN	0.000	0.003	-1	1	1.000	0.045	0.000	0.064	0.064	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.028	0.001	NaN	0.001	0.000	-1	5	0.911	0.045	0.000	0.606	0.606	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.001	NaN	0.000	0.003	-1	5	1.000	0.006	0.000	0.558	0.558	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	112.647	0.0	300	0.001	0.001	NaN	0.561	0.024	200.738	200.927	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.141	0.002	300	0.006	0.0	0.824	0.472	0.012	0.298	0.298	See	See